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Performance in science on the Minnesota 
Comprehensive Assessments-Series II 
for students in grades 5 and 8 



This study of statewide performance 
on the 2009/10 Minnesota state science 
assessment in grades 5 and 8 found that 
most of the variation in test scores was 
associated with demographic differences 
among students. Average performance 
differed by gender, eligibility for free or 
reduced-price lunch, special education 
status, and race/ethnicity. 

Policymakers in Minnesota have made as- 
sessing and improving student science per- 
formance a priority (Minnesota High Tech 
Association 2010a, b). Minnesota has sup- 
ported several statewide initiatives to promote 
science, technology, engineering, and math 
(STEM) — including a grant from the National 
Governors Association to increase science 
learning opportunities, align K-12 STEM 
education requirements with postsecondary 
workplace expectations, improve the quality 
and quantity of STEM teachers, benchmark 
standards, and identify best practices in STEM 
education (National Governors Association 
2007). To gauge student progress toward the 
state’s academic science standards, the Min- 
nesota Department of Education introduced 
the Minnesota Comprehensive Assessments- 
Series II (MCA-II) science assessment in 2008. 

Regional Educational Laboratory Midwest 
responded to a request by the Minnesota 



Department of Education to study elemen- 
tary and middle school science achievement. 
The current study addresses three research 
questions: 

• How does student achievement on the 
2009/10 MCA-II science assessment in 
grades 5 and 8 differ by student demo- 
graphic characteristics? 

• How does schoolwide achievement on 
the 2009/10 MCA-II science assess- 
ment in grades 5 and 8 relate to school 
characteristics? 

• To what extent do school characteristics 
explain differences in student achievement 
on the 2009/10 MCA-II science assess- 
ment in grades 5 and 8, after accounting 
for the influence of student characteristics? 

This study used data for 51,510 grade 5 students 
in 786 schools and 52,421 grade 8 students in 
469 schools. Five student demographic charac- 
teristics were considered: gender, eligibility for 
free or reduced-price lunch, special education 
status, race/ethnicity, and prior-year academic 
achievement. The school characteristics exam- 
ined were based on student composition and 
teacher composition. The school characteristics 
based on student characteristics were the per- 
centages of female students, students eligible 
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for free or reduced-price lunch, students 
identified as limited English proficient, stu- 
dents identified for special education services, 
and White students. The school characteristics 
based on teacher characteristics were the aver- 
age years of teacher experience, the percentage 
of full-time equivalent credits 1 taught by teach- 
ers with a master’s or doctoral degree, and the 
average student-teacher ratio. 

Students’ levels of science achievement de- 
pended on their demographic characteristics 
for both the grade 5 and grade 8 assessment: 

• Students who were not identified for spe- 
cial education services scored higher than 
students who were. 

• Students who were not eligible for free or 
reduced-price lunch scored higher than 
students who were. 

• White students scored higher than stu- 
dents of other racial/ethnic groups. 

• Male students scored higher than female 
students. 

Eight of the nine school characteristics ex- 
amined were related to schoolwide science 
achievement: 

• Science achievement tended to be lower 
in schools with higher percentages of 
students eligible for free or reduced-price 
lunch, limited English proficient students, 
students identified for special education 
services, and non-White students. 

• Science achievement tended to be higher 
in schools that had more experienced 



teachers, higher percentages of teachers 
with an advanced degree, larger student- 
teacher ratios, and higher levels of prior- 
year academic achievement. 2 

• Science achievement tended to be higher 
in schools with a higher proportion of fe- 
male students in grade 8, but this relation- 
ship was not observed in grade 5. 

About 80 percent of the variation in students’ 
scores (79 percent in grade 5 and 84 percent 
in grade 8) was due to differences among 
students within schools; differences between 
schools accounted for the remaining variation. 
However, after accounting for student-level 
characteristics, the school characteristics 
examined explained less than 3 percent of the 
variation between schools (2.1 percent in grade 
5 and 2.7 percent in grade 8). For both grades, 
after accounting for student characteristics, 
science achievement tended to be higher in 
schools with a smaller percentage of students 
eligible for free or reduced-price lunch and a 
larger percentage of White students. 

Specifically, the following results were evident 
for both grades: 

• A 1 percentage point increase in the per- 
centage of White students was associated 
with an estimated increase in science as- 
sessment scores of 3.24 percentage points 
in grade 5 and 2.31 percentage points in 
grade 8. 

• A 1 percentage point increase in the 
percentage of students eligible for free or 
reduced-price lunch was associated with 
an estimated decrease in science assess- 
ment scores of 2.33 percentage points in 
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grade 5 and 2.65 percentage points in 
grade 8. 

None of the school characteristics based on 
teacher composition examined in this study 
were related to student science achievement 
after other student and school characteristics 
were accounted for. 

This report provides Minnesota policymakers 
with insights into factors related to science 
achievement, as measured by the MCA-II. 

The analyses examine patterns of achievement 
but do not explain why the patterns occur. 

The findings identify demographic subgroups 
in Minnesota (such as non-White students 
and students eligible for free or reduced-price 
lunch) that could benefit from more intensive 
support in science. Findings are consistent 
with research by Stewart (2008) and Konstan- 
topoulos (2006). 

Differences in science achievement between 
limited English proficient students and those 
who are not limited English proficient could 



not be examined because of high levels of 
missing test data for limited English proficient 
students. Further, the study could examine 
only a limited set of school characteristics 
based primarily on student and teacher 
composition. The state might consider exam- 
ining school characteristics related directly to 
science education, such as course and program 
offerings, science teacher performance, sci- 
ence teacher preparation and experience, and 
resources to support science instruction. 



1. Full-time equivalent credits represent the 
amount of time per week a teacher is reported in 
a teaching assignment. 

2. The student-teacher ratio is the average number 
of students per teacher in each school. In this 
study, student achievement tended to increase 
with the number of students per teacher. How- 
ever, this could have been due to schools with 
more struggling students decreasing their class 
sizes to improve student achievement. 
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This study 
of statewide 
performance 
on the 2009/10 
Minnesota 
state science 
assessment in 
grades 5 and 8 
found that most 
of the variation 
in test scores was 
associated with 
demographic 
differences among 
students. Average 
performance 
differed by 
gender, eligibility 
for free or 
reduced-price 
lunch, special 
education status, 
and race/ethnicity. 



WHY THIS STUDY? 

Concerned about future U.S. economic com- 
petitiveness and prosperity, policymakers 
nationwide and in Minnesota have renewed their 
interest in science, technology, engineering, 
and math (STEM; National Academy of Science 
2005; Obama 2010; U.S. Department of Educa- 
tion 2006; Minnesota High Tech Association 
2010a, b). To promote STEM fields, Minnesota 
has supported several statewide initiatives, 
including a web portal to link business resources 
with the needs of STEM educators (www.get- 
stem-mn.com) and a website where parents and 
students can learn about the link between STEM 
fields and careers (www.mn-stem.com). In 2007, 
Minnesota received a grant from the National 
Governors Association to fund programs to 
increase science learning opportunities, align 
K-12 STEM education requirements with post- 
secondary workplace expectations, increase the 
quality and quantity of STEM teachers, bench- 
mark standards, and identify best practices in 
STEM education (National Governors Associa- 
tion 2007). 

In 2008, the Minnesota Department of Educa- 
tion introduced the Minnesota Comprehensive 
Assessments-Series II (MCA-II) science as- 
sessments to gauge student progress toward the 
state’s academic science standards (see box 1 for 
a description of the MCA-II and appendix A on 
science standards). The assessment is adminis- 
tered every spring in grades 5 and 8, as well as 
once during grades 9-12 after a student com- 
pletes a life science course. To help policymakers 
target resources to improve science achievement 
overall and to reduce achievement gaps, the Min- 
nesota Department of Education asked Regional 
Educational Laboratory Midwest to examine 
MCA-II science assessment scores in grades 5 
and 8 and to determine how the scores vary by 
student and school characteristics (see box 2 for 
definition of key terms). 
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BOX 1 

Minnesota Comprehensive 
Assessment-Series II 

Since 2008, the Minnesota Depart- 
ment of Education has administered 
the Minnesota Comprehensive 
Assessment-Series II (MCA-II) 
science assessment every spring 
in grades 5 and 8 and once in high 
school after a student completes a 
life science course. The assessment is 
aligned to the Minnesota K-12 Aca- 
demic Standards in Science, which 
details students’ expected knowledge 
in general science, physical science, 
earth and space science, and life 
science (see appendix A). Depending 
on the grade, each of these strands 
accounts for 22-32 percent of the 
assessment, with content specific to 
each grade (see Minnesota Depart- 
ment of Education 2009a for details 
of the assessment). The assessment is 
based on Webb’s four depth of knowl- 
edge levels (Webb 1999): 

• Recall of a fact, definition, term, 
or process. 

• Skill/concept: mental processing 
beyond recall such as how to ap- 
proach a problem. 

• Strategic thinking: reasoning, 
planning, or use of evidence. 



• Extended thinking: complex rea- 
soning, planning, and thinking. 

MCA-II is web-based, allowing for 
simulations of classroom experi- 
ments and manipulations of real- 
world visual representations (Minne- 
sota Department of Education 2007). 

MCA-II assessments have demon- 
strated strong validity and reliability. 
Content validity is shown by the use 
of experts to write the items based 
on standards, and scoring validity by 
agreement among raters on scoring 
of extended-response items, good 
item fit using item response theory 
analyses, and principal component 
analyses showing individual factors 
for each standard. Alpha reliability 
coefficients are 0.82 for the grade 5 
test and 0.86 for the grade 8 test. 1 

The grade 5 assessment has a maxi- 
mum score of 41 raw points derived 
from items related to 9-11 scenarios. 2 
The items consist of 34 multiple- choice 
or figure-based items (1 point each), 
two short-response or figural 3 items 
(2 points each), and one extended- 
response item (3 points). The grade 8 
assessment is similarly structured: 37 
multiple-choice and figural-response 
items, four 2-point short-response or 
figural-response items, and one 3 -point 
extended-response item based on 7-9 



scenarios (48 raw points; Minnesota 
Department of Education 2008). 

The Minnesota Department of Educa- 
tion converts the raw scores (points 
earned for answering items correctly) 
to scale scores using an item response 
theory model, which provides a 
consistent metric for forms of the 
assessment that vary in difficulty. 

The MCA-II scores for a given grade 
and subject are scaled from X01 
to X99, where X is the grade being 
examined. In the current study, grade 
5 students have an average score 
of 548.89 points (with a standard 
deviation of 13.52 points), and grade 
8 students have an average score of 
848.73 points (standard deviation of 
10.38 points). The construction of the 
assessment does not allow grade 5 
scores to be compared directly with 
grade 8 scores. 

Notes 

1. Coefficients of 0.80 are generally seen 
as the threshold of acceptable reli- 
ability (Nunnally 1978). 

2. A sample scenario: “A student is 
growing a garden. In her garden, she 
is growing many types of fruits and 
vegetables, including bell peppers. 
Which of the following factors affect- 
ing the growth of pepper plants is 
biotic?” 

3. For example, a figural item could 
require a student to construct or 
manipulate a graph. 



Research questions 

The current study addresses three research 
questions: 

• How does student achievement on the 

2009/10 MCA-II science assessment in grades 
5 and 8 differ by student demographic char- 
acteristics? 



• How does schoolwide achievement on the 
2009/10 MCA-II science assessment in grades 
5 and 8 relate to school characteristics? 

• To what extent do school characteristics 
explain differences in student achievement 
on the 2009/10 MCA-II science assessment in 
grades 5 and 8, after accounting for the influ- 
ence of student characteristics? 
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BOX 2 

Key terms 

Between-school variance. The propor- 
tion of variation in student test scores 
explained by school characteristics 
(such as percentage of students eli- 
gible for free or reduced-price lunch 
or student-teacher ratio) and school- 
wide policies and practices. 

Eligibility for free or reduced-price 
lunch. A commonly used indicator of 
low-income status. Students are eli- 
gible for free lunch if their household 
income is at or below 130 percent of 
the poverty level and for reduced- 
price lunch if the household income 
is 130-185 percent of the poverty 
level (U.S. Department of Agriculture 
2010 ). 

Full-time equivalent credits. The 
proportion of time per week an 
individual teacher is reported in a 
teaching assignment. This variable is 
used instead of the number of teach- 
ers, which can be misleading because 
some teachers work part-time and 
some serve in multiple or nonaca- 
demic roles (Minnesota Department 
of Education 2006). For more infor- 
mation, see Lewit and Baker (1997). 

Limited English proficient students. 
Students for whom English is not 
their first language and who cannot 
perform typical class work in English 
(McCandless, Rossi, and Daugherty 
1996). 

Minnesota Comprehensive 
Assessment-Series II (MCA-II) scale 
scores. Raw scores converted to a 
common scale. Grade 5 scale scores 



range from 501 to 599 and grade 8 
scale scores range from 801 to 899. 

For more information on the MCA-II 
science test, see box 1. 

Multilevel modeling. A regression 
model used to analyze nested data 
(such as students within schools). It 
accounts for similarities among indi- 
viduals (such as students) who belong 
to the same clusters (such as schools) 
by including a cluster-specific term 
(random effect) that is shared by all 
individuals in the cluster. This ap- 
proach makes it possible to examine 
variance in science achievement 
attributable to differences within 
schools and between schools. It also 
estimates the relationship between 
student achievement and student- 
and school-level characteristics and 
accounts for the uncertainty in these 
estimates (that is, correctly calculates 
the standard errors of the estimates). 

National Assessment of Educational 
Progress. Also known as the nation’s 
report card, an ongoing assessment 
of the largest representative sample of 
students from U.S. schools in 12 sub- 
jects (including math, reading, and 
science). Results are reported nation- 
ally and by district and state for the 
entire sample and by demographic 
characteristic (National Center for 
Education Statistics 2011a). 

Pearson correlation coefficient. A 
measure of the linear association be- 
tween two variables, ranging from -1 
(perfect negative relationship, mean- 
ing that as one variable increases, 
the other decreases), to 1 (perfect 
positive relationship, meaning that 
as one variable increases, the other 



increases). The coefficient equals 
0 if the two variables are linearly 
unrelated. 

Prior-year academic achievement. For 
students, a composite of a student’s 
MCA-II math and reading test scores 
from the previous grade. For schools, 
the school average of composite math 
and reading student test scores. 

School-level characteristics. Charac- 
teristics of schools such as the per- 
centage of female students or average 
years of teacher experience in a given 
school. The same school-level charac- 
teristics were examined for grades 5 
and 8. (See table B2 in appendix B for 
the characteristics examined in this 
study.) 

Special education students. Students 
identified as needing special educa- 
tion services and who may or may 
not be receiving them. 

Standard deviation. A measure of the 
amount of variation or spread in a 
distribution around the average. The 
greater the variance, the greater the 
standard deviation. 

Standardized average difference. The 
difference between two group aver- 
ages and the pooled standard devia- 
tion. The current study used Cohen’s 
d to calculate the standardized aver- 
age difference (or effect size), by tak- 
ing the difference between two aver- 
age scores (such as male and female 
student scores) divided by the pooled 
standard deviation to show the 
extent of the variation, or “spread,” 
among scores of the two groups. 
Comparing the average differences 
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BOX 2 (CONTINUED) 

Key terms 

with the amount of variation in the 
two groups indicates whether the 
differences are meaningful. Higher 
Cohen’s d values reflect larger aver- 
age differences and less variation 
among the scores in the two groups. 
Cohen categorized standardized 
average differences of roughly 0.20 as 
small, differences of roughly 0.50 as 
medium, and differences of roughly 
0.80 as large. Lipsey’s (1989) review 
of education interventions gener- 
ally confirms this categorization 
(small = 0.15, medium = 0.45, and 
large = 0.90). 



Student-level characteristics. Char- 
acteristics of students, such as race/ 
ethnicity or gender. The same student 
characteristics were examined for 
grade 5 and grade 8. (See table B2 in 
appendix B for the characteristics 
examined in this study.) 

Variance explained. The propor- 
tion of variance (within or between 
schools) in student scores explained 
by the characteristics included in a 
multilevel model, which indicates 
the degree of influence that one 
characteristic (such as the percentage 



of female students in a school) or 
a group of characteristics (such 
as a school’s student composition 
characteristics) has in explaining 
differences in student scores. The 
proportion of variance explained can 
range from 0 percent to 100 percent, 
with higher percentages indicating 
a stronger relationship between one 
characteristic (or a group of charac- 
teristics) and student scores. 

Within-school variance. The amount 
of variation in student scores among 
students in the same school. 



This study used data for 51,510 grade 5 students 
in 786 schools and 52,421 grade 8 students in 
469 schools. Five student demographic charac- 
teristics were considered: gender, eligibility for 
free or reduced-price lunch, special education 
status, race/ethnicity, and prior-year academic 
achievement. The school characteristics exam- 
ined were based on student composition and 
teacher composition. The school characteristics 
based on student characteristics were the per- 
centages of female students, students eligible for 
free or reduced-price lunch, students identified 
as limited English proficient, students identi- 
fied for special education services, and White 
students. The school characteristics based on 
teacher characteristics were the average years 
of teacher experience, the percentage of full- 
time equivalent credits 1 taught by teachers with 
a master’s or doctoral degree, and the average 
student-teacher ratio. 



Prior research on science achievement — 

Minnesota and nationally 

Although little research specifically addresses sci- 
ence achievement in Minnesota, results from the 
2009 National Assessment of Educational Progress 



(NAEP) in science show that 43 percent of grade 4 
students in Minnesota scored proficient or higher 
on the 2009 NAEP in science compared with 32 
percent nationwide, and 40 percent of grade 8 stu- 
dents scored proficient or higher compared with 
29 percent nationwide. 

However, some subgroups in Minnesota per- 
formed no better on the NAEP than their 
counterparts nationally. In particular, there were 
no statistically significant differences between 
Minnesota and national results in the percent- 
ages of students scoring proficient or higher for 
Black students, Hispanic students, and special 
education students in both grades 4 and 8 and for 
students eligible for free or reduced-price lunch 
in grade 4 (National Assessment of Educational 
Progress 2009). 

Both nationally and in Minnesota, there are 
gaps in science achievement on the NAEP across 
student subgroups (U.S. Department of Education 
2009a, b). NAEP science scores in grades 4 and 
8 were higher for male students than for female 
students, for White students than for Black and 
Hispanic students, for students not eligible for free 
or reduced-price lunch than for eligible students, 
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and for students identified for special education 
services than for students not identified. 

NAEP results have indicated the importance of 
student-level characteristics in describing differ- 
ences in science achievement, but the MCA-II 
science assessment is a new test that includes 
different items and gauges students’ knowledge 
against different standards. Therefore, students 
may perform differently on the MCA-II than on 
the NAEP. 

In 2009/10, fewer than half of Minnesota grade 5 
students (46.0 percent) and grade 8 students (47.9 
percent) achieved proficiency on the MCA-II 
science assessment, a rate lower than for math 
or reading (Minnesota Department of Education 
2011). The Minnesota Department of Education 
website has an interactive tool that allows users 
to see gaps in MCA-II science scores and profi- 
ciency levels across student demographic charac- 
teristics. However, the tool does not indicate the 
magnitude of the differences across subgroups, 
nor does it examine the relationship between 
school-level characteristics and student science 
achievement, which are the focus of the current 
study. 



Prior research on characteristics related 
to science achievement 

Using 1992 National Education Longitudinal 
Study data, Konstantopoulos (2006) found that 
roughly 80 percent of student achievement in 
science is explained by differences in the char- 
acteristics of students within schools, and the 
other 20 percent by differences between schools. 
Other studies using different data also found 
that more of the variation in student achieve- 
ment is explained by characteristics of students 
within schools than by differences between 
schools (Battistich et al. 1995; Lee and Bryk 
1989). 

Several studies have examined which 
school characteristics are related to student 



The current study will 
provide information 
for Minnesota on the 
relationship between 
key student and school 
characteristics and 
science performance 



achievement. Some 
researchers have found 
that schools with disad- 
vantaged student popu- 
lations (those with high 
percentages of racial/ 
ethnic minority stu- 
dents and students from 
low-income households) 
tend to have lower average student science 
and math achievement (Arnold and Kaufman 
1992; Hogrebe, Kyei-Blankson, and Zou 2008; 
Konstantopolous 2006). By contrast, Stewart 
(2008) found that once student characteristics 
—household income, race/ethnicity, student 
effort, and positive peer relationships — are 
taken into account, a school’s size, location, 
percentage of racial/ethnic minority students, 
and percentage of students from low-income 
households do not significantly relate to student 
achievement, measured by a composite of math, 
science, English, and history grades (see also 
Hanushek 1997). 



Research examining teacher characteristics 
has found that degree attainment (Hogrebe et 
al. 2008) and teaching experience (Greenwald, 
Hedges, and Laine 1996) relate positively with 
measures of student achievement, while student- 
teacher ratios relate negatively (Finn and Achilles 
1990; Konstantopolous 2006; Krueger and Whit- 
more 2001; McGiverin, Gilman, and Tillitski 1989; 
Nyhan and Alkadry 1999). The evidence, however, 
is inconsistent; other studies find that smaller 
class sizes are associated with lower academic 
performance. 2 

Overall, the research on the influence of school 
characteristics on student achievement, which 
has not been specific to Minnesota, is inconclu- 
sive. The current study will provide state-specific 
information for Minnesota on the relationship 
between key student and school characteristics 
and science performance. (Box 3 summarizes the 
study data and methods; details are in appendixes 
B and D-F.) 
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BOX 3 

Data, samples, and methods 

Data. The Minnesota Department of 
Education provided data on student- and 
school-level scale scores on the Minne- 
sota Comprehensive Assessment-Series 
II (MCA-II) science assessment, student 
demographic characteristics, and 
teacher characteristics for public, char- 
ter, and magnet schools for 2009/10. 1 
Duplicate student records or records 
missing one of the key analytic variables 
were deleted (13.2 percent of grade 5 
students and 12.0 percent of grade 8 
students; see appendix B for details). 
Missing data were related primarily to 
prior-year academic achievement, espe- 
cially prior-year MCA-II math scores. 2 

The analytic samples retained more 
than 70 percent of students in each sub- 
group (see tables Cl and C3 in appen- 
dix C), except for limited English pro- 
ficient students. 3 The analytic samples 
retained 40.6 percent of limited English 
proficient students in grade 5 and 45.0 
percent in grade 8. Data tended to be 
missing for prior-year math scores, af- 
fecting 56.8 percent of grade 5 students 
and 49.3 percent of grade 8 students. 

For both grades, there were statisti- 
cally significant differences between 
the limited English proficient students 
who had complete data on the current 
year’s science scores and the prior year’s 
reading and math scores and those 
with missing data. In addition, there 
were statistically significant differences 
between students who had missing data 
on race/ethnicity and identification as 
a special education student for grade 5 
and on race/ethnicity for grade 8 and 
students who had complete data. These 
differences could make findings based 
on student-level data for limited Eng- 
lish proficient students misleading, so 
they are not discussed in this report. 4 



Student analytic samples. The analy- 
ses included 51,510 students in grade 
5 and 52,421 students in grade 8 in 
2009/10. For both grades, approxi- 
mately half the students were female, 
less than 35 percent were eligible for 
free or reduced-price lunch, less than 
14 percent were identified for special 
education services, and more than 78 
percent were White (table 1). 

School analytic samples. The analyses 
included 786 schools for grade 5 and 
469 schools for grade 8. On average, 
in schools enrolling grade 5 or grade 8 
students, approximately half the stu- 
dents were female, less than 44 per- 
cent were eligible for free or reduced- 
price lunch, less than 11 percent were 
limited English proficient, less than 
16 percent were identified for special 
education services, and more than 71 
percent were White (table 2). Teachers 
in these schools had an average of ap- 
proximately 14 years of teaching expe- 
rience, less than half the credits were 



taught by teachers with an advanced 
degree (master’s or doctoral) and 
there were an average of 16 students 
per teacher. 

Methods. For the first research ques- 
tion, student science achievement in 
2009/10 was compared across student 
characteristics. '/Tests, adjusted for 
student clustering within schools, were 
used to determine whether differences 
in average science achievement by stu- 
dent characteristics were statistically 
significant. Since the analytic data set 
has more than 50,000 observations, 
even small differences could be statisti- 
cally significant, so standardized aver- 
age differences were calculated, using 
Cohen’s d (Cohen 1988). 

For the second research question, 
Pearson correlations were calculated 
between average schoolwide science 
achievement and school-level charac- 
teristics based on student composi- 
tion 5 and teacher characteristics. 



TABLE 1 

Distribution of students in the analytic samples by student characteristic 



Student 


Grade 5 students 


Grade 8 students 


characteristic 


Number 


Percent 


Number 


Percent 


All students 


51,510 


100.00 


52,421 


100.00 


Gender 


Male 


26,223 


50.9 


26,548 


50.6 


Female 


25,287 


49.1 


25,873 


49.4 


Free or reduced-price lunch 


Eligible 


17,811 


34.6 


16,313 


31.1 


Not eligible 


33,699 


65.4 


36,108 


68.9 


Special education status 


Identified 


6,855 


13.3 


5,735 


10.9 


Not identified 


44,655 


86.7 


46,686 


89.1 


Race/ethnicity 


White 


40,506 


78.6 


41,964 


80.1 


Non-White a 


11,004 


21.4 


10,457 


20.0 



Note: Percentages might not sum to 100 because of rounding. 

a. For racial/ethnic minority breakdown, see tables Cl and C3 in appendix C. 

Source: Authors' analysis based on data from the Minnesota Department of Education. 
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BOX 3 (CONTINUED) 

Data, samples, and methods 

For the final research question, the study 
analyzed how much of the variation in 
individual student science achievement 
can be explained by differences among 
students in the same school (within- 
school variance) and how much by 
differences between the characteristics 
of schools (between-school variance). 
Also analyzed was how school-level 
characteristics relate to student achieve- 
ment after accounting for the relation- 
ship between student characteristics and 
student achievement. Five multilevel 
regression models were estimated. First, 
a model without student or school char- 
acteristics was estimated to examine the 
extent to which differences in science 



scores can be attributed to differences 
within or across schools (model 1). Then, 
characteristics were entered successively: 
student-level demographic data (model 
2), student-level prior achievement 
(model 3), school-level student composi- 
tion (model 4), and school-level teacher 
characteristics (model 5). 

Characteristics were entered in groups 
to explore how much of the remain- 
ing differences in science achievement 
were explained by different types of 
student- and school-level character- 
istics. In addition, regression coeffi- 
cients were estimated to discern which 
school characteristics are significantly 



TABLE 2 

Descriptive characteristics of schools included in the analytic samples 



School characteristic 


Grade 5 


Grade 8 


Percentage of female students 


48.6 


48.4 




(3.99) 


(4.88) 


Percentage of students eligible for free or 


43.7 


40.0 


reduced-price lunch 


(23.52) 


(21.26) 


Percentage of limited English proficient students 


10.2 


5.7 




(16.29) 


(13.19) 


Percentage of students identified for special 


14.3 


15.2 


education services 


(5.71) 


(10.95) 


Percentage of White students 


71.6 


78.0 




(28.31) 


(26.20) 


Teachers' years of teaching experience 


14.2 


13.5 




(4.00) 


(3.66) 


Percentage of full-time equivalent credits taught 


49.7 


43.1 


by teachers with an advanced degree 3 


(23.05) 


(21.56) 


Student-teacher ratio 


15.7 


15.7 




(4.34) 


(5.35) 


Prior-year academic achievement (MCA-II 


-0.1 


-0.2 


standardized math and reading composite score) 


(0.46) 


(0.45) 



Note: Percentages might not sum to 100 because of rounding. Numbers in parentheses are standard 
deviations. School characteristics were based on all students enrolled in each school in 2009/10 (not 
just grade 5 and grade 8 students), except for school average prior-year academic achievement, 
which is the average of the standardized grade 4 or grade 7 MCA-II math and reading composite 
score only for students in the grade 5 and grade 8 analytic samples. Negative values for prior-year 
academic achievement indicate that the average of schools based on the analytic sample was lower 
than the average of schools based on all students enrolled. 

a. Calculated for each school by dividing the numbers of school-level full-time equivalent credits 
taught by teachers with a master's or doctoral degree by the sum of the numbers of school-level 
full-time equivalent credits taught by all teachers and then averaging across schools. 

Source: Authors' analysis based on data from the Minnesota Department of Education. 



associated with individual science per- 
formance (see appendix D and table E2 
in appendix E). Regression coefficients 
represent the relationship between 
school characteristics and student sci- 
ence scores, after taking into account 
the proportion of the relationship 
explained by other characteristics. 

Notes 

1. The results for private school students and 
students who take alternative assessments 
typically are not reported by the Minne- 
sota Department of Education and are not 
included in this study. 

2. Less than 9 percent of students in each 
grade were missing prior-year academic 
achievement data, less than 4 percent were 
missing school characteristics, and less 
than 3 percent were missing demographic 
data or MCA-II science scores. 

3. For grade 5, the subgroups with the largest 
percentages of students excluded from all 
analyses were limited English proficient 
(40.6 percent of data retained), Asian/ 
Pacific Islander (70.8 percent retained), 
identified as special education (76.5 percent 
retained), and Black (78.5 percent retained) 
students (see table Cl in appendix C). For 
grade 8, the largest percentages of students 
excluded from all analyses were limited 
English proficient (45.0 percent of data 
retained), Hispanic (70.6 percent retained), 
special education identified (75.3 percent 
retained), and Black (77.8 percent retained) 
students (see table C3 in appendix C). 

4. Data on school-level percentages of 
limited English proficient students were 
not affected by the missing data (for more 
detail, see tables F3-F6 in appendix F). 

5. To reflect the global environment of the 
school, school composition characteristics 
were based on all students in a school, 
not just those in grades 5 and 8. There is 
less than a 10 percentage point difference 
in the school composition characteristics 
when calculated for all students (as in the 
current analysis) or when calculated for 
just grade 5 or grade 8 students. See table 
F2 in appendix F for a comparison across 
all school composition characteristics for 
both grade 5 and grade 8 students. 
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STUDY FINDINGS 

This section presents the findings for each re- 
search question. 

Differences in science achievement by 
student demographic characteristics 

There were statistically significant differences 
in science achievement for each grade in all four 
subgroup comparisons (figures 1 and 2 and table 1; 
table FI in appendix F shows the /-statistics and 
p-values associated with the subgroup differences). 

For both grades 5 and 8, medium to large differ- 
ences were found in favor of students not identi- 
fied for special education services, White students, 
and students not eligible for free or reduced-price 
lunch. 3 Students who were not identified for 
special education services scored 10.49 percent- 
age points (effect size = 0.80) higher than students 
identified for special education services in grade 
5 and 10.05 percentage points (effect size = 1.02) 
higher in grade 8. 4 Although gender gaps were 



statistically significant for both grades, the dif- 
ferences were substantively small. Male students 
scored 0.85 percentage point (effect size = 0.06) 
higher than female students in grade 5 and 1.35 
percentage points (effect size = 0.13) higher in 
grade 8. Students not eligible for free or reduced- 
price lunch scored 9.77 percentage points (effect 
size = 0.77) higher than eligible students in grade 
5 and 7.37 percentage points (effect size = 0.75) 
higher in grade 8. White students scored 10.46 
percentage points (effect size = 0.82) higher than 
non-White students in grade 5 and 7.25 percentage 
points (effect size = 0.73) higher in grade 8. 



Differences in science achievement 
by school characteristics 

Schoolwide science achievement tended to be 
higher in schools with a lower percentage of 
students eligible for free or reduced-price lunch, 
limited English proficient students, and students 
identified for special education services; a higher 
percentage of White students; and a higher 
prior-year average MCA-II math and reading 



FIGURE 1 

Grade 5 average Minnesota Comprehensive Assessment-Series II science scores, by student demographic 
characteristics, 2009/10 




Overall Male Female Eligible Not Identified Not White Non-White a Black Hispanic Asian/ Native 
(n = 51,510) (n = 26,223) (n = 25,287) (n = 17,81 1) eligible (n = 6,855) identified (n = 40,506) (n = 1 1 ,004) (n = 4,570) (n = 2,677) Pacific American 

(n = 33,699) (n = 44,655) Islander (n = 1,100) 

(n = 2,657) 

Gender Free or Special education Race/ethnicity 

reduced-price lunch 

Note: This figure reports unadjusted scale-score averages. See appendix F for details on statistical significance, 
a. Aggregates Black, Hispanic, Asian/Pacific Islander, and Native American students. 

Source: Authors' analysis based on data from the Minnesota Department of Education. 
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FIGURE 2 

Grade 8 average Minnesota Comprehensive Assessment-Series II science scores, by student demographic 
characteristics, 2009/10 




Overall Male Female Eligible Not Identified Not White Non-White a Black Hispanic Asian/ Native 
(n = 52, 421) (n = 26, 548) (n = 25, 873) (n = 16,313) eligible (n = 5,735) identified (n = 41 ,964) (n = 1 0,457) (n = 4,054) (n = 2,433) Pacific American 

(n = 36,108) (n = 46,686) Islander (n = 1,029) 

(0 = 2,941) 

Gender Free or Special education Race/ethnicity 

reduced-price lunch 

Note: This figure reports unadjusted scale-score averages. See appendix F for details on statistical significance, 
a. Aggregates Black, Hispanic, Asian/Pacific Islander, and Native American students. 

Source: Authors' analysis based on data from the Minnesota Department of Education. 



TABLE 1 

Mean and standardized difference for grades 5 and 8 Minnesota Comprehensive Assessment-Series II 
science scores, 2009/10 



1 Student characteristic 


Grade 5 


Grade 8 


Gender 


Mean difference between male and female students 


0.85*** 


1.35*** 


Standardized difference 


0.06 


0.13 


Free or reduced-price lunch 


Mean difference between eligible and not eligible students 


-9 77 *** 


-7.37*** 


Standardized difference 


-0.77 


-0.75 


Special education status 


Mean difference between students identified for services and not identified 


-10.49*** 


-10.05*** 


Standardized difference 


-0.80 


- 1.02 


Race/ethnicity 


Mean difference between White and non-White students 


10.46*** 


7.25*** 


Standardized difference 


0.82 


0.73 



*** Statistically significant at the 0.001 level. 

Note: This table reports unadjusted scale-score averages in the direction of the first subgroup listed for each characteristic for 51,510 students in grade 5 
and 52,421 students in grade 8. Standardized differences were calculated using Cohen's d (see box 2). Significance test results come from two-tailed f-tests 
comparing subgroup averages using a two-level regression model that adjusts for student clustering within schools (see table FI in appendix F). 

Source: Authors' analysis based on data from the Minnesota Department of Education. 
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achievement score. These results were statisti- 
cally significant and consistent across grades 5 
and 8. For grade 8, schoolwide science achieve- 
ment tended to be higher in schools with a higher 
percentage of female students, but for grade 5 the 
relationship was not statistically significant. The 
relationship between each of the school character- 
istics based on student composition and average 
school science achievement was consistent with 
the average differences in student achievement for 
each related demographic characteristic. The ex- 
ception was for gender in grade 8, as mentioned. 5 

Schoolwide teacher characteristics were also re- 
lated to schoolwide science achievement. School- 
wide science achievement tended to be higher in 
schools with more experienced faculty, with a 
higher percentage of teachers with an advanced 
degree, and with higher school student-teacher 
ratios. 6 These results were statistically significant 
and consistent across grades. 



Variation in science achievement within and between schools 

Consistent with Konstantopoulos (2006), this 
study found that the variation in performance on 



the MCA-II science assessment was due largely to 
differences among students within schools. More 
than 75 percent of the variation in student science 
scores (79.2 percent in grade 5 and 84.1 percent in 
grade 8) was due to differences among students 
within schools (see appendix E for more details). 
Differences between schools accounted for the 
remaining variation (20.8 percent in grade 5 and 
16.0 percent in grade 8). After accounting for the 
influence of school-level characteristics based on 
student composition, differences in school charac- 
teristics explained less than 3 percent of the varia- 
tion in science scores between schools. The school 
characteristics based on student composition that 
were significantly associated with science achieve- 
ment were the percentage of students eligible for 
free or reduced-price lunch and the percentage of 
White students. 

Between-school variation. Achievement differ- 
ences between schools were explained primarily 
by differences in the student population. Together, 
student demographics and prior-year academic 
achievement explained 79.7 percent of the varia- 
tion in science scores between schools in grade 5 
and 82.1 percent in grade 8. After accounting for 



TABLE 2 

Correlation between schoolwide Minnesota Comprehensive Assessment-Series II science scores and school 
characteristics for grades 5 and 8, 2009/10 





Pearson correlation 
coefficient 


School characteristic 


Grade 5 
(n = 786) 


Grade 8 
(n = 469) 


Percentage of female students 


0.02 


0.26*** 


Percentage of students eligible for free or reduced-price lunch 


-0.74*** 


-0.75*** 


Percentage of limited English proficient students 3 


-0.45*** 


-0.34*** 


Percentage of students identified for special education services 


-0.21*** 


-0.46*** 


Percentage of White students 


0.65*** 


0.58*** 


Years of teaching experience 


0.22*** 


0.22*** 


Percentage of full-time equivalent credits taught by teachers with an advanced degree 


0.15*** 


0.20*** 


Student-teacher ratio 


0.22*** 


0.23*** 


Prior year's Minnesota Comprehensive Assessment-Series II math and reading composite score 


0.86*** 


0.88*** 



*** Statistically significant at the 0.001 level. 

a. Includes results for limited English proficient students because school-level percentages for this student subgroup were not affected by the missing data. 
Source: Authors' analysis based on data from the Minnesota Department of Education. 




STUDY LIMITATIONS 



11 



the influence of individual student characteristics, 
school characteristics based on student composi- 
tion and teacher composition together explained 
2.1 percent of the differences in science achieve- 
ment between schools in grade 5 and 2.7 percent 
in grade 8. 

However, only a limited set of school character- 
istics was available in the data. A portion of the 
variance between schools (18.3 percent in grade 
5 and 15.2 percent in grade 8) is not explained 
by variables in this study and might be due to 
unobserved student and school characteristics (for 
example, science teachers’ quality of teaching) and 
random error. 7 

Associations between student science achievement 
and school characteristics based on student com- 
position. After the influence of individual student 
characteristics was accounted for, two school char- 
acteristics had a statistically significant association 
with achievement in both grades: the percent- 
age of students eligible for free or reduced-price 
lunch and the percentage of White students (see 
table E2 in appendix E). 8 The percentages of female 
students, 9 students identified for special education 
services, and average prior-year academic achieve- 
ment were not significant predictors of science 
achievement. 

If all other factors remained the same, the results 
suggest the following: 

• Percentage of students eligible for free or 
reduced-price lunch. A 1 percentage point 
increase in the percentage of students eligible 
for free or reduced-price lunch in a school was 
associated with an estimated decrease in stu- 
dent science scores of 2.3 percentage points in 
grade 5 and 2.7 percentage points in grade 8. 

• Percentage of White students. A 1 percentage 
point increase in the percentage of White 
students was associated with an estimated 3.2 
percentage point increase in student science 
achievement in grade 5 and a 2.3 percentage 
point increase in grade 8. 



Associations between stu- 
dent science achievement 
and teacher characteris- 
tics. After the influence of 
individual student char- 
acteristics and school- 
level characteristics based 
on student composition 
were accounted for, none 
of the teacher-related 
school characteristics 
examined in this study 
(years of experience, 
percentage of teachers 
with an advanced degree, 
student-teacher ratio) was related to grade 5 or 
grade 8 student science achievement. 



After the influence 
of individual student 
characteristics was 
accounted for, two 
school characteristics 
had a statistically 
significant association 
with achievement in both 
grades: the percentage 
of students eligible for 
free or reduced-price 
lunch and the percentage 
of White students 



STUDY LIMITATIONS 

The current study has several limitations. First, the 
associations of student and school characteristics 
with achievement are based on correlation analy- 
sis, so causality cannot be inferred. The analysis 
shows how student science performance differs by 
student characteristic (such as gender and race/ 
ethnicity) and school characteristic (such as the 
percentage of students eligible for free or reduced- 
price lunch). It cannot show, however, that being 
female, for instance, “causes” a student to perform 
at a certain level. Rather, these findings provide a 
snapshot of overall patterns of achievement and 
identify demographic subgroups that are at risk for 
poor performance in science and that could benefit 
from more intensive support in science education. 

Second, private school students and students who 
took alternative assessments are not included in 
the analysis, and therefore the conclusions apply 
only to the public, charter, and magnet school 
students who took the MCA-II in 2009/10. 

Third, the study included only nine school char- 
acteristics, none directly related to the science 
instruction. For example, the teacher-related 
school characteristics included information based 
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on all teachers, not science teachers specifically. 
Minnesota is developing a system for capturing 
course data; future research could use these data 
to look for an association between coursework and 
other teacher- and school-related characteristics 
and student achievement on the MCA-II. 



Fourth, the school characteristics based on student 
composition were for an entire school, not just for 
grade 5 or grade 8. The results differed somewhat 
when student percentages and schoolwide percent- 
ages were examined for a given grade instead, but 
most differences were less than 5 percent (see table 
F2 in appendix F). 



After accounting for 
student characteristics, 
school factors explained 
little of the variation 
between schools in 
science achievement 
in 2009/10 



Fifth, students with missing 
data for either student or school 
characteristics were not included 
in the analyses. In most cases, the 
students were removed because 
of missing data on prior-year 
academic achievement. Students 
with missing data differ in some 
respects from those without miss- 
ing data, so the results cannot be generalized to 
students who were not included in the analyses. For 
some subgroups, such as limited English proficient 
students and students identified for special educa- 
tion services, more than 20 percent of students 
were excluded from the analyses because of miss- 
ing data. Further analysis (reported in appendix F) 
indicated that limited English proficient students 
who were excluded from the study due to missing 
data differed significantly from students who were 
included on the MCA-II and demographic charac- 
teristics. Thus, conclusions about limited English 
proficient students cannot be drawn, and findings 
for these students were not discussed in this report. 



CONCLUSIONS 

This report aims to provide Minnesota policy- 
makers with insight into factors related to sci- 
ence achievement, as measured by the MCA-II. 
Although differences in science achievement by 
student demographic characteristics have been 



well documented, the current study is the first to 
look comprehensively at student characteristics 
and scores on the MCA-II science assessment. 
Using 2009/10 data, the study found medium to 
large statistically significant differences in average 
science achievement for grades 5 and 8 favor- 
ing students not identified for special education 
services, students not eligible for free or reduced- 
price lunch, and White students. Small differences 
were found favoring male students. The results re- 
lated to gender, eligibility for free or reduced-price 
lunch, and race/ethnicity are consistent with those 
of previous research using national samples (Gon- 
zales et al. 2008; Konstantopoulos 2006; National 
Assessment of Educational Progress n.d.). 

Beyond student characteristics, the limited set 
of school factors examined here do not seem to 
influence science achievement. While correlation 
analysis showed associations between schoolwide 
science achievement and school characteristics 
based on student and teacher composition, these 
relationships mostly disappeared after accounting 
for student characteristics. Moreover, after ac- 
counting for student characteristics, school factors 
explained little of the variation between schools in 
science achievement in 2009/10, which is consistent 
with findings from Stewart (2008) and Konstanto- 
poulos (2006). Although some studies have found 
significant relationships between teacher charac- 
teristics and student achievement (Finn and Achil- 
les 1990; Krueger and Whitmore 2001; McGiverin 
et al. 1989; Nyhan and Alkadry 1999), the current 
study was consistent with Konstantopoulos (2006) 
in finding no significant relationship between the 
student-teacher ratio and achievement. In the 
current study, teacher characteristics explained less 
than 1 percent of the differences in school achieve- 
ment between schools, and the teacher characteris- 
tics were not statistically significant. 

Future work could examine the effects of school 
characteristics specifically related to science 
instruction, such as course and program offer- 
ings, science teacher performance, science teacher 
preparation and experience, and resources to sup- 
port science instruction. 
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APPENDIX A 

MINNESOTA K-12 ACADEMIC 
STANDARDS IN SCIENCE 

This appendix lists the Minnesota K-12 academic 
standards in science, as enumerated in Minnesota 
Department of Education (2009a). 

Strand 1: Nature of Science and Engineering 

• Substrand 1: The Practice of Science 

• Standard 1. Understandings about science 

• Standard 2. Scientific inquiry and 
investigation 

• Substrand 2: The Practice of Engineering 

• Standard 1. Understandings about 
engineering 

• Standard 2. Engineering design 

• Substrand 3: Interactions among Science, Tech- 
nology, Engineering, Mathematics, and Society 

• Standard 1. Systems 

• Standard 2. Careers and contributions in 
science and engineering 

• Standard 3. Mutual influence of science, 
engineering, and society 

• Standard 4. The role of math and technol- 
ogy in science and engineering 



Strand 2: Physical Science 

• Substrand 1: Matter 

• Standard 1. Properties and structure ofmatter 

• Standard 2. Changes in matter 

• Substrand 2: Motion 

• Standard 1. Describing motion 

• Standard 2. Forces 

• Substrand 3. Energy 

• Standard 1. Kinds of energy 

• Standard 2. Energy transformations 

• Substrand 4. Human Interactions with Physi- 
cal Systems 

• Standard 1. Interaction with the 
environment 



Strand 3: Earth and Space Science 

• Substrand 1. Earth Structure and Processes 

• Standard 1. Plate tectonics 

• Standard 2. Earth’s changing surface 

• Standard 3. Rock sequences and Earth 
history 

• Substrand 2. Interdependence within the 
Earth System 

• Standard 1. Sources and transfer of 
energy 

• Standard 2. Weather and climate 

• Standard 3. Materials cycles 

• Substrand 3. The Universe 

• Standard 1. Solar system motion 

• Standard 2. Formation of the solar system 

• Standard 3. Age, scale, and origin of the 
universe 

• Substrand 4. Human Interactions with Earth 
Systems 

• Standard 1. Interaction with the 
environment 



Strand 4: Life Science 

• Substrand 1. Structure and Function in Living 
Systems 

• Standard 1. Levels of organization 

• Standard 2. Cells 

• Substrand 2. Interdependence among Living 
Systems 

• Standard 1. Ecosystems 

• Standard 2. Flow of energy and matter 

• Substrand 3. Evolution in Living Systems 

• Standard 1. Reproduction 

• Standard 2. Variation 

• Standard 3. Biological evolution 

• Substrand 4. Human Interactions with Living 
Systems 

• Standard 1. Interaction with the 
environment 

• Standard 2. Health and disease 
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APPENDIX B 
DATA PROCEDURES 

This appendix describes the study’s data sources 
and analysis procedures. 



Data sources 

For restricted-use data, a formal request was 
placed with the Minnesota Data Compliance 
Department; a data-sharing agreement under the 
Family Educational Rights and Privacy Act was 
created and signed to allow access to the data. 

The data were transmitted through a secure site 
maintained by the Minnesota Department of 
Education. To protect student confidentiality, data 
were provided with unique student identifying 
codes rather than student names. These student 
identifiers were used to link student records in the 
demographic data and assessment files. Only au- 
thorized research personnel had access to the data. 
Public-use files were obtained from the Minnesota 
Department of Education website. 

Restricted-use datafiles. The Minnesota Depart- 
ment of Education provided two restricted-use stu- 
dent data files (with grade 5 and grade 8 informa- 
tion combined; figure Bl). The first file contained 
student demographic data— gender, eligibility for 
free or reduced-price lunch, limited English pro- 
ficiency status, special education status, and race/ 
ethnicity — collected in the fall of a given school 
year and updated when a student had a change in 
free or reduced-price lunch eligibility, limited Eng- 
lish proficiency status, or special education status. 
If a student’s status changed during the year, the 
most recent status was used in the analysis. These 
data were collected by the state as part of the 
Minnesota Automated Reporting Student System 
(MARSS). The second data file contained students’ 
Minnesota Comprehensive Assessment-Series II 
(MCA-II) scores and demographic information 
(collected each spring with the administration of 
MCA-II science assessment). Pearson Assessments 
collects MCA-II student assessment data annually 
and provides the data to the Minnesota Depart- 
ment of Education. 



Public-use datafiles. Public-use data files con- 
tained school characteristics based on student and 
teacher composition (see figure Bl). 

• School-level student demographic data in- 
cluded the percentages of female students, stu- 
dents eligible for free or reduced-price lunch, 
limited English proficient students, students 
identified for special education services, and 
White students. 

• School-level student achievement data in- 
cluded current year (2009/10) MCA-II science 
scores and prior years’ MCA-II composite 
math and reading scores. 

• School-level teacher data included teacher 
experience (average years of teaching for 
teachers at each school), degree attainment 
(percentage of full-time equivalent credits 
taught by teachers with advanced degrees), 
and the student-teacher ratio (average num- 
ber of students per teacher). 

The public-use data files at the school level in- 
cluded all schools that had either a grade 5 or a 
grade 8. A few schools had both grades. 



Data cleaning and merging 

This section discusses how the two student record 
files were merged and how missing data were 
handled. 

Student assessment and demographic files. Two 

student record files (containing assessment data 
and demographic data from the MARSS) were 
merged using unique student identifiers and 
district and school codes (see figure Bl). Before 
the two files were merged, duplicate and missing 
demographic and test score data in the assessment 
file had to be resolved. In some cases, students 
had multiple rows in the assessment file — for 
example, because they had two sets of test scores. 
When there were multiple rows in the assessment 
file for a single student record, discrepancies were 
resolved using the following procedures: 
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FIGURE B1 

Process for obtaining and merging Minnesota Department of Education data files 
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Merged grade 5 files based on school ID — 
all student- and school-level characteristics for 2009/10 



Merged grade 8 files based on school ID — 
all student- and school-level characteristics for 2009/10 



Note: Shaded boxes indicate points where data files were used to conduct data analyses. 



• If all rows in the assessment file for the same 
student had demographic information, the 
demographic information was compared with 
the demographic data in the MARSS file. Data 
from rows with discrepant information were 
dropped. The demographic data in MARSS 
were used as the standard because those data 
are collected by schools and school districts 
and verified for accuracy by the state. In 
2009/10, across both grades, fewer than 0.05 
percent of cases in the assessment file had 
duplicate rows deleted because of discrepan- 
cies in demographic data. 

• If all rows in the assessment file for the same 
student had demographic information and 
there was no discrepancy, the row with the 
highest MCA-II scale score was kept. In 



2009/10 across both grades, fewer than 0.05 
percent of cases had duplicate rows deleted 
after keeping the highest score. 

• If some rows in the assessment file for the 
same student had missing demographic data, 
the row with available data that matched 
demographic data in the MARSS was kept. 
Discrepancies and missing data for gender 
and race/ethnicity were examined before the 
other demographic characteristics. In 2009/10 
across both grades, 0.02 percent of students 
had discrepancies for gender, and 0.51 percent 
of students were missing data on gender in 
one source; 0.37 percent of students had dis- 
crepancies for race/ethnicity, and 0.60 percent 
of students were missing data on race/ethnic- 
ity in one source. 
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Missing data. Analyses were performed on 
complete cases only. After student demographic 
and assessment files were merged, any student 
cases missing one of the key variables — MCA-II 
science scale scores, student demographic in- 
formation, student prior-year math and reading 
scale scores, or school covariates — were deleted 
manually. 

Table B1 shows the total number of students in 
the files received from the Minnesota Department 
of Education after duplicate values were recon- 
ciled, as well as the number of students excluded 
because of missing data. Less than 2 percent of 
students were missing MCA-II science scores; 
most of these students were identified for special 



education services or took alternative assessments 
(such as the Minnesota Test of Academic Skills, 
a test administered to students with significant 
cognitive impairments) or both. MCA-II sci- 
ence score data were not imputed for students 
who took alternative assessments because these 
students were deemed better served by alternative 
assessments or for the remaining students (less 
than 2 percent of the total excluded) because the 
percentage of missing data was so small. Next, 
students who were missing data for at least one 
demographic characteristic (less than 1 percent 
of the samples) were deleted. Demographic data 
such as gender were not imputed at the student 
level because there is no logical way to impute 
someone’s gender. 



TABLE B1 

Number of students with missing data and percentage of the original sample for grades 5 and 8, 2009/10 





Grade 5 


Grade 8 


Characteristic 


Number of 
students with 
missing data 


Percentage of 
original sample 
(n = 59,311) 


Number of 
students with 
missing data 


Percentage of 
original sample 
(n = 59,564) 


Student characteristics 


Minnesota Comprehensive Assessment-Series II 
science scores 


941 


1.6 


1,002 


1.7 


Student demographic data (gender, free or reduced- 
price lunch eligibility, special education status, limited 
English proficiency status, and race/ethnicity) 


286 


0.5 


493 


0.8 


Prior-year academic achievement 


5,005 


8.4 


3,760 


6.3 


School characteristics 


Student-related (percentage of females, students 
eligible for free or reduced-price lunch, students 
identified for special education services, limited 
English proficient students, and White students) 


0 


0.00 


83 


0.14 


School average prior-year academic achievement 3 


1,617 


2.7 


2,106 


3.5 


Teacher-related (years of experience, teacher degree 
attainment, 13 and student-teacher ratio) 


157 


0.26 


76 


0.13 


Students whose data were deleted 3 


7,801 


13.2 


7,143 


12.0 


Analytic sample for questions 1 and 3 


51,510 


86.9 


52,421 


88.0 



Note: Components might not sum to totals because of rounding. 

a. Prior-year (grade 4 for grade 5 and grade 7 for grade 8) school average Minnesota Comprehensive Assessment-Series II math and reading composite 
score. 

b. The percentage of full-time equivalent credits taught by teachers with a master's or doctoral degree. 

c. Total deleted is not equal to the sum of previous rows because some students were missing data for multiple characteristics. 

Source: Authors' analysis based on data from the Minnesota Department of Education. 
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Prior-year academic achievement data on the 
MCA-II math and reading assessments (a com- 
posite score) in grades 4 and 7 were missing for 
8.4 percent of students in grade 5 and 6.3 percent 
of students in grade 8, respectively. Missing data 
were not imputed because more than half the 
students (2,705 of 5,005) with missing prior-year 
academic achievement scores were also missing 
either their math or reading scores for the current 
year, a key covariate for improving the reliability 
of the imputation procedure. 

After students with missing demographic data 
were excluded, the student-level files were merged 
with the school-level files. This consolidated data 
file included each student and the characteristics 
of a given student’s school. Some schools did not 
have data for certain characteristics. Because 
missing school-level data across all school charac- 
teristics (other than prior-year academic achieve- 
ment) for both grades 5 and 8 never totaled more 
than 2.4 percent, complete case analysis was used 
instead of data imputation. Less than 4 percent of 
students were missing school average prior-year 
academic achievement (2.73 percent in grade 5 
and 3.54 percent in grade 8), less than 0.30 percent 
of students were missing teacher-related charac- 
teristics (0.26 percent in grade 5 and 0.13 percent 
in grade 8), and less than 0.20 percent of students 
were missing student-related school characteristics 
(0.00 percent in grade 5 and 0.14 percent in grade 
8). Overall, the total percentage of students ex- 
cluded because of missing data was 13.15 percent 
in grade 5 and 11.99 percent in grade 8. 



Data characteristics and special coding 

The statistical analysis programs SAS and SPSS 
(Statistical Package for the Social Sciences) were 
used to code and aggregate raw data to create the 
characteristics shown in table B2. This section dis- 
cusses the data on characteristics that were ready 
to analyze as obtained and those that required 
manipulation before they could be analyzed. 

Student achievement. Current-year (2009/10) grade 
5 and grade 8 MCA-II science scale scores needed 



no data manipulation when acquired from the 
Minnesota Department of Education. The study 
used scores only from students who completed 
the MCA-II science assessment. Students who 
took alternative state tests (such as the Minnesota 
Test of Academic Skills) were excluded from the 
analyses on MCA-II science outcomes because the 
two forms could not be equated. 

Since math and reading test scores are likely to be 
highly correlated, a composite measure of prior- 
year academic achievement was calculated and 
used. For students within each grade, the prior 
year MCA-II math and reading scale scores were 
standardized separately (by subtracting the grade 
mean and dividing by grade standard deviation), 
added together, and then restandardized to form 
the composite score. 

School characteristics. For each school, the Min- 
nesota Department of Education website provides 
data on the school characteristics shown in table 
B2. With the exception of the prior-year school av- 
erage math and reading composite score, the school 
characteristics are for an entire school, not just for 
grade 5 or 8. The prior-year school average achieve- 
ment was calculated by standardizing the average 
of the prior-year MCA-II math and reading scores 
of current year grade 5 or grade 8 students included 
in the analytic samples, summing the standardized 
math and reading scores separately for each grade, 
and then standardizing them again. 

The Minnesota Department of Education provided 
teacher degree attainment measured as the num- 
ber of full-time equivalent (FTE) credits taught 
by teachers with each type of degree (less than a 
bachelor’s degree, bachelor’s degree, master’s degree, 
or doctoral degree) at the school level. FTE credits 
represent the amount of time per week an individual 
is reported in a teaching assignment. To obtain a 
single continuous characteristic, the FTE credits 
taught by teachers with a master’s or doctoral degree 
were divided by the total number of FTE credits for a 
school across all degree types. This new characteris- 
tic represented the percentage of FTE credits taught 
by teachers with an advanced degree. Master’s level 
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TABLE B2 

Data elements, coding, and use 



Data element 


Data coding 


Data use 


Unique student identifier 


Unique student number assigned by the state 
(no coding needed) 


Unique record ID 


Race/ethnicity 


5 = White; 4 = Black; 3 = Hispanic; 

2 = Asian/Pacific Islander; 1 = Native American 


Demographic characteristic analysis 


White/non-White a 


0 = White; 1 = non-White 


Demographic characteristic analysis; 
multilevel model 


Gender 


0 = male; 1 = female 


Demographic characteristic analysis; 
multilevel model 


Free or reduced-price lunch 
indicator 


1 = eligible for free or reduced-price lunch; 

0 = not eligible for free or reduced-price lunch 


Demographic characteristic analysis; 
multilevel model 


Special education status 


1 = special education identification; 

0 = no special education identification 


Demographic characteristic analysis; 
multilevel model 


Limited English proficiency 


1 = limited English proficient; 

0 = not limited English proficient 


Multilevel model 


Grade 5 and grade 8 MCA-II 
scale scores 


Scale scores for math, reading, and science, used as 
provided 


Demographic characteristic analysis; 
multilevel model 


Unique school identifier 


Combined the school number and district number to 
create a unique identifier 13 


Correlation analysis; multilevel model 


School name 


No coding for this characteristic, just a display of each 
school name 


Correlation analysis; multilevel model 


Gender 


Percentage of female students 


Correlation analysis; multilevel model 


Free or reduced-price lunch 
indicator 


Percentage of students eligible for free or reduced- 
price lunch 


Correlation analysis; multilevel model 


Special education status 


Percentage of students identified for special education 
services 


Correlation analysis; multilevel model 


Limited English proficiency 


Percentage of limited English proficient students 


Correlation analysis; multilevel model 


Race/ethnicity 


Percentage of White students 


Correlation analysis; multilevel model 


School achievement 


School average scale scores on the Minnesota 
Comprehensive Assessment-Series II math and reading 
tests (two scale scores combined to form a composite 
score) and average science scale scores 


Correlation analysis; multilevel model 


Teacher experience 


Average years of total teaching experience for all 
teachers in each school 


Correlation analysis; multilevel model 


Teacher degree attainment 


For each school, the percentage of full-time equivalent 
credits taught by teachers with a master's or doctoral 
degree relative to all full-time equivalent credits taught 
at each school 


Correlation analysis; multilevel model 


Student-teacher ratio 


Average student-teacher ratio at each school 


Correlation analysis; multilevel model 



a. Constructed characteristic. Non-White includes Black, Hispanic, Asian/Pacific Islander, and Native American students. 

b. Districts assign numbers to their schools, but they are not unique across districts. Therefore, combining the district and school numbers created a unique 
school ID. 

Source: Minnesota Department of Education public-use school-level data. 



and higher was chosen because the goal was to 
represent higher degree levels among teachers (most 
teachers had earned a bachelor’s degree). Student 



performance was hypothesized to relate to teachers’ 
education attainment based on research by Hogrebe 
et al. (2008) and Goldhaber and Brewer (1996). 
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APPENDIX C 

STUDENT DEMOGRAPHIC AND SCHOOL 
CHARACTERISTICS FOR GRADES 5 AND 8 

This appendix includes tables showing the sample 
size of demographic subgroups and correlations 
among school characteristics for grades 5 and 8. 



TABLE Cl 

Grade 5 sample size and percentage of students relative to the original student population, by student 
characteristic, 2009/10 



Demographic characteristic and subgroup 


Number of students 

Analytic sample Original population 3 


Percentage of 
original sample 


All students 


51,510 


59,311 


86.9 


Gender 


Male 


26,223 


30,784 


85.2 


Female 


25,287 


28,784 


87.9 


Free or reduced-price lunch 


Eligible 


17,811 


22,386 


79.6 


Not eligible 


33,699 


36,633 


92.0 


English proficiency 


Limited 


2,025 


4,991 


40.6 


Not limited 


49,485 


54,311 


91.1 


Special education status 


Identified 


6,855 


8,959 


76.5 


Not identified 


44,655 


50,060 


89.2 


Race/ethnicity 


White 


40,506 


43,971 


92.1 


Non-White 


11,004 


15,048 


73.1 


Black 


4,570 


5,825 


78.5 


Hispanic 


2,677 


4,232 


63.3 


Asian/Pacific Islander 


2,657 


3,751 


70.8 


Native American 


1,100 


1,240 


88.7 



a. For each demographic characteristic, the sum of the subgroup sample sizes (for example, male students plus female students) does not sum to the total 
number of students (59,311 in this example) because some students had missing values for some demographic characteristics. 

Source: Authors' analysis based on data from the Minnesota Department of Education. 
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TABLE C2 

Correlation among school characteristics for the 786 schools included in the grade 5 analytic sample 



School 

characteristic 


Percentage Percentage 
White female 


Percentage 
eligible 
for free or 
reduced- 
price lunch 


Percentage 
identified 
for special 
education 
services 


Percentage 

limited 

English 

proficient 


Teacher 

experience 


Teacher 

degree 


Student- 

teacher 

ratio 


Math and 
reading 
composite 
score 3 


Percentage White 


1.00 


















Percentage female 


-0.03 


1.00 
















Percentage eligible for 
free or reduced-price 
lunch 


-0.73*** 


0.04 


1.00 














Percentage identified 
for special education 




















services 


-0.02 


-0.27*** 


0.28*** 


1.00 












Percentage limited 
English proficient 


-0.78*** 


-0.01 


0.59*** 


-0.15*** 


1.00 










Teacher experience 


0.30*** 


-0.04 


-0.16*** 


0.15*** 


-0.25*** 


1.00 








Teacher's degree 


-0.12*** 


-0.01 


-0.26*** 


-0.26*** 


0.12*** 


0.13*** 


1.00 






Student-teacher ratio 


0.11*** 


0.10* 


-0.32*** 


-0.35*** 


-0.11*** 


0.01 


0.25*** 


1.00 




Math and reading 
composite score 3 


0.58*** 


0.02 


-0.75*** 


-0.28*** 


-0.40*** 


0.18*** 


0.19*** 


0.21*** 


1.00 



* Statistically significant at the 0.05 level; *** statistically significant at the 0.001 level, 
a. Composite score on the Minnesota Comprehensive Assessment-Series II reading and math tests. 
Source: Authors' analysis based on data from the Minnesota Department of Education. 
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TABLE C3 

Grade 8 sample size and percentage of students relative to the original student population, by student 
characteristic, 2009/10 



Demographic characteristic and subgroup 


Number of students 

Analytic sample Original population 3 


Percentage of the 
original sample 


All students 


52,421 


59,564 


88.0 


Gender 


Male 


26,548 


30,211 


87.9 


Female 


25,873 


28,913 


89.5 


Free or reduced-price lunch 


Eligible 


16,313 


20,151 


81.0 


Not eligible 


36,108 


38,898 


92.8 


English proficiency 


Limited 


1,577 


3,502 


45.0 


Not limited 


50,844 


55,547 


91.5 


Special education status 


Identified 


5,735 


7,618 


75.3 


Not identified 


46,686 


51,431 


90.8 


Race/ethnicity 


White 


41,964 


45,504 


92.2 


Non-White 


10,457 


13,545 


77.2 


Black 


4,054 


5,212 


77.8 


Hispanic 


2,433 


3,444 


70.6 


Asian/Pacific Islander 


2,941 


3,696 


79.6 


Native American 


1,029 


1,193 


86.3 



a. For each demographic characteristic, the sum of the subgroup sample sizes (for example, male students plus female students) does not sum to the total 
number of students (59,564 in this example) because some students had missing values for some demographic characteristics. 

Source: Authors' analysis based on data from the Minnesota Department of Education. 
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TABLE C4 

Correlation among school characteristics for the 469 schools included in the grade 8 analytic sample 



School 

characteristic 


Percentage Percentage 
White female 


Percentage 
eligible 
for free or 
reduced- 
price lunch 


Percentage 
identified 
for special 
education 
services 


Percentage 

limited 

English 

proficient 


Teacher 

experience 


Teacher 

degree 


Student- 

teacher 

ratio 


Math and 
reading 
composite 
score 3 


Percentage White 


1.00 


















Percentage female 


0.05 


1.00 
















Percentage eligible for 
free or reduced-price 
lunch 


-0.75*** 


-0.17*** 


1.00 














Percentage identified 
for special education 




















services 


-0.17*** 


-0.58*** 


0.35*** 


1.00 












Percentage limited 
English proficient 


-0.73*** 


0.05 


0.57*** 


-0.07 


1.00 










Teacher experience 


0.36*** 


0.03 


-0.21*** 


-0.11* 


-0.26*** 


1.00 








Teacher's degree 


-0.12* 


0.05 


-0.20*** 


-0.04 


0.09* 


0.08 


1.00 






Student-teacher ratio 


-0.03 


0.26*** 


-0.20*** 


-0.33*** 


0.03 


-0.03 


0.23*** 


1.00 




Math and reading 
composite score 3 


0.45*** 


0.33*** 


-0.69*** 


-0.53*** 


-0.25*** 


0.19*** 


0.25*** 


0.31*** 


1.00 



* Statistically significant at the 0.05 level; *** statistically significant at the 0.001 level, 
a. Composite score on the Minnesota Comprehensive Assessment-Series II reading and math tests. 
Source: Authors' analysis based on data from the Minnesota Department of Education. 




APPENDIX D. DATA ANALYSIS 



23 



APPENDIX D 
DATA ANALYSIS 

This appendix describes the data analysis for each 
research question. 



Research question 1 

For each grade, Minnesota Comprehensive 
Assessment-Series II (MCA-II) science assess- 
ment score averages were calculated for the entire 
sample and by demographic characteristic (such as 
gender and race/ethnicity). Differences in average 
MCA-II scale scores between subgroups were also 
calculated. For a more nuanced look at the magni- 
tude of differences in averages, standardized aver- 
age differences or effect sizes (Cohen’s d; Cohen 
1988) were also reported. T-tests were conducted 
to examine the statistical significance of the dif- 
ferences in average scores between demographic 
subgroups. Reported f-test statistics and p-values 
were corrected for clustering, using a two -level 
multilevel model with a random intercept and 
with the demographic characteristic of interest as 
the only predictor. 



Research question 2 

Pearson correlations were calculated between 
school-level average performance on the MCA-II 
science assessment and school characteristics 
based on student and teacher composition. 



Research question 3 

Two-level multilevel models, with students nested 
in schools, were run to predict student-level 
achievement on the MCA-II science assessment 
for grades 5 and 8 separately. Stata and the lme4 
software package in R were used for the analyses 
(Bates, Maechler, and Bolker 2011). 

For each grade, five models were estimated in 
three stages. In the first stage, an unconditional 



model (which did not include any student- or 
school-level characteristics) was estimated to 
establish the amount of within- and between- 
school variance in student science achievement 
(model 1). In the following stages, student- and 
school-level characteristics were added in blocks 
to discern how much variation student- and 
school-level characteristics explained. In the 
second stage, model 2 included student gender, 
eligibility for free or reduced-price lunch, special 
education status, limited English proficiency 
status, and race/ethnicity (White/non-White). 
Model 3 added prior-year academic achievement. 
In the third stage, school-level characteristics 
were added. Model 4 added school-level stu- 
dent characteristics: the percentages of female 
students, students eligible for free or reduced- 
price lunch, limited English proficient students, 
students identified for special education services, 
and White students, plus prior-year school aver- 
age composite math and reading achievement. 
Model 5 added school-level teacher characteris- 
tics: teacher experience, teacher degree attain- 
ment, and student-teacher ratio. Models in the 
third stage estimated the relationship between 
school-level characteristics and science achieve- 
ment, after accounting for the influence of 
student-level characteristics. 

The study used grand-mean centering for all vari- 
ables (continuous and categorical) in the analysis 
for research question 3 because its principal aim 
was to estimate the influence of school charac- 
teristics (level-2 predictors) on student science 
achievement after accounting for the influence of 
student characteristics. Grand-mean centering 
of both student- and school-level characteristics 
established meaningful intercepts and slopes that 
allowed level-2 predictors to account for between- 
school differences (Enders and Tofighi 2007). 
Categorical characteristics were dummy-coded 
using one of the categories as a reference category 
(for example, 0 = male and 1 = female) prior to 
centering. 
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The final specification of the model was as follows: 

Level 1: students within schools 

Yjj — Ttgj + iTtpjXpjj + C q 

where Y q = either grade 5 or grade 8 score on the 
MCA-II science assessment of student i in school;; 
n Qj = estimated average MCA-II science score 
for school; adjusted for student characteristics 
Xpiji TTpj = estimated association between MCA-II 
science score and X pi j adjusted for other student 
characteristics; X pij = pth student characteristic 
(see table B2 in appendix B) for student i in school 
; (grand-mean centered); and = random error 
term for student i in school;. 



Level 2: schools 

”0j “ Poo ~t LfioqWqj + Kqj 
n pj = Ppo 

where (3 00 = expected average school mean MCA- 
II science score (in grade 5 or 8) across all schools; 
(3 0lJ = expected average difference in school mean 
MCA-II science score associated with one unit 
difference in W qj , adjusted for other school-level 
characteristics; W qj = school characteristic q for 
school; (grand-mean centered); r 0j = school-level 
random error term; and (3^ 0 = expected average 
difference in MCA-II science score associated 
with one unit difference in X pq adjusted for other 
student characteristics. 



APPENDIX E. VARIANCE DECOMPOSITION FROM MULTILEVEL MODELS 



25 



APPENDIX E 

VARIANCE DECOMPOSITION 
FROM MULTILEVEL MODELS 

This appendix describe the amount of variation in 
student achievement explained by five multilevel 
models, beginning with a baseline model with no 
student or school characteristics (model 1) and 
successively adding student characteristics (mod- 
els 2 and 3) and school characteristics (models 4 
and 5). See Table El for details. 

• Model 1: overall variation in achievement. 
Most of the variation in student science scores 



in 2009/10 (79.2 percent in grade 5 and 84.1 
percent in grade 8) was due to differences 
among students in the same schools (within- 
school variance). 10 Differences between 
schools accounted for the remaining variation 
in student science achievement (20.8 percent 
in grade 5 and 16.0 percent in grade 8). 

• Models 2 and 3: differences in achievement be- 
tween schools explained by student characteris- 
tics. The differences between schools in science 
achievement were due primarily to differences 
in the kinds of students who attended differ- 
ent schools. Individual student demographics, 



TABLE El 

Multilevel models and the percentages of between-school variance in grades 5 and 8 student science 
achievement explained 





Block of 




Estimated percentage of 
between-school variance 


Model 


characteristics added 


Characteristics included in each block 


Grade 5 


Grade 8 


1 


Unconditional model 


— 


20.76 


15.95 








Estimated percentage of between- 
school variance explained 








Grade 5 


Grade 8 


2 


Student demographic 
characteristics 


• Gender 

• Eligibility for free or reduced-price lunch 

• Limited English proficiency status 

• Special education status 

• Race/ethnicity (White/non-White) 


57.5 


58.0 


3 


Student prior- 
year academic 
achievement 


• Grade 4 or grade 7 Minnesota Comprehensive 
Assessment-Series II (MCA— II) math and reading 
composite score 


22.2 


24.2 


Between-school variance explained by models 2 and 3 


79.7 


82.1 


4 


School characteristics 


• School student demographic variables 3 


2.1 


2.7 




based on student 
composition 


• School average grade 4 or grade 7 MCA-II math 
and reading composite score 






5 


School characteristics 
based on teacher 
composition 


• Teacher years of experience 

• Degree attainment 

• Student-teacher ratio 


-0.01 b 


0.00 


Between-school variance explained by models 4 and 5 


2.1 


2.7 


Total between-school variance explained by model 5 


81.7 


84.9 



/Vote: The with in-school variance for model 1 is not shown because the focus of this section is on differences between schools. Components might not sum 
to totals because of rounding. 

a. School characteristics based on student composition included in model 4 are the percentages of female students, students eligible for free or reduced- 
price lunch, limited English proficient students, students identified for special education services, and White students. 

b. When fixed effects are added to a multilevel model, small decreases (less than 5 percent) and even negative variance in the proportion of variance ex- 
plained can occur by chance. Larger decreases might signal possible model misspecification (Snijders and Bosker 1999). 

Source: Authors' analysis based on data from the Minnesota Department of Education. 
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added in model 2, and student prior-year 
academic achievement, added in model 3, 
together explained 79.7 percent of the variation 
in grade 5 and 82.1 percent in grade 8. 

• Models 4 and 5: differences in achievement be- 
tween schools explained by school characteristics. 
After the influence of student characteristics 



(entered in models 2 and 3) was accounted for, 
school characteristics explained a small portion 
of the overall variation in performance on the 
science assessment. School characteristics based 
on student composition, added in model 4, 
and teacher characteristics, added in model 5, 
together explained 2.1 percent of the variation 
in grade 5 and 2.7 percent in grade 8. 



TABLE E2 

Grades 5 and 8 multilevel model regression coefficients and standard errors for the full model (model 5) 



1 Characteristic 


Grade 5 


Grade 8 


Student characteristics 


Female 


-2.00*** 

(0.07) 


-2.62*** 

(0.05) 


Eligible for free or reduced-price lunch 


-0.94*** 

(0.09) 


-0.55*** 

(0.07) 


Identified for special education services 


-1.10*** 

(0.11) 


-0.20* 

(0.09) 


Limited English proficient 


-3.02*** 

(0.23) 


-1.75*** 

(0.19) 


White 


2.29*** 

(0.11) 


1.62*** 

(0.08) 


Prior-year academic achievement 3 


9.49*** 

(0.04) 


7.85*** 

(0.03) 


School characteristics 


Percentage of female students 


5.27 

(2.97) 


2.27 

(2.43) 


Percentage of students eligible for free or reduced-price lunch 


-2.33* 

(1.02) 


-2.65** 

(.91) 


Percentage of students identified for special education services 


3.65 

(2.58) 


-1.68 

(1.40) 


Percentage of limited English proficient students' 3 


4.38*** 

(1.17) 


2.88*** 

(1.07) 


Percentage of White students 


3.24*** 

(0.83) 


2.31*** 

(0.72) 


School average teacher experience 


0.02 

(0.03) 


-0.02 

(0.03) 


Percentage of full-time equivalent credits taught by teachers with advanced degrees 


0.00 

(0.01) 


0.01 

(0.00) 


Student-teacher ratio 


0.05 

(0.03) 


0.00 

(0.02) 


School average prior-year academic achievement 1 


0.19 

(0.38) 


-0.57 

(0.34) 



* Statistically significant at the 0.05 level; ** statistically significant at the 0.01 level; *** statistically significant at the 0.001 level. 

Note: Numbers in parentheses are standard errors. 

a. Prior-year (grade 4 or grade 7) student Minnesota Comprehensive Assessment-Series II (MCA— II) math and reading composite score. 

b. Because of missing data, more than 50 percent of limited English proficient students were removed from the sample. Those that remained were not 
representative of all limited English proficient students, which could lead to biased results for the comparison of limited English proficient and non-limited 
English proficient students. 

c. Prior-year (grade 4 or grade 7) school average MCA-II math and reading composite score. 

Source: Authors' analysis based on data from the Minnesota Department of Education. 
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APPENDIX F p-values associated with subgroup differences for 

ADDITIONAL ANALYSES research question 1. Table F2 shows the differences 

between student- and school-level data. Tables F3- 
This appendix presents the results of additional F6 provide more detail on missing data for limited 

analyses. Table FI shows the f-statistics and English proficient students. 



TABLE FI 

Grades 5 and 8 significance test results for research question 1 





Grade 5 




Grade 8 




Student characteristic 


Mean difference 


p-value 


Mean difference 


p-value 


Female 


-0.85 


< 0.001 


-1.39 


< 0.001 




(0.11) 




(0.09) 




Eligible for free or reduced-price 


-7.41 


< 0.001 


-5.82 


< 0.001 


lunch 


(0.12) 




(0.10) 




Identified for special education 


-9.92 


< 0.001 


-9.47 


< 0.001 


services 


(0.16) 




(0.13) 




White 


8.10 


< 0.001 


6.01 


< 0.001 




(0.15) 




(0.12) 




Race/ethnicity pairwise comparisons 


Native American-Asian/Pacific 


-4.78 


< 0.001 


-3.49 


< 0.001 


Islander 


(0.49) 




(0.38) 




Native American-Flispanic 


-0.23 


1.000 


0.43 


1.000 




(0.48) 




(0.38) 




Native American-Black 


1.82 


0.001 


3.21 


< 0.001 




(0.46) 




(0.36) 




Native American-White 


-8.67 


< 0.001 


-5.66 


< 0.001 




(0.42) 




(0.33) 




Asian/Pacific Islander-Flispanic 


4.55 


< 0.001 


3.92 


< 0.001 




(0.35) 




(0.27) 




Asian/Pacific Islander-Black 


6.60 


< 0.001 


6.70 


< 0.001 




(0.31) 




(0.24) 




Asian/Pacific Islander-White 


-3.90 


< 0.001 


-2.17 


< 0.001 




(0.26) 




(0.20) 




Flispanic-Black 


2.05 


< 0.001 


2.78 


< 0.001 




(0.31) 




(0.25) 




Flispanic-White 


-8.44 


< 0.001 


-6.09 


< 0.001 




(0.26) 




(0.21) 




Black-White 


-10.49 


< 0.001 


-8.87 


<0.001 




(0.22) 




(0.17) 





Note: Numbers in parentheses are standard errors. The mean differences are estimated using a two-level multilevel model, which adjusts for clustering. The 
p-values are based on a two-tailed f-test of difference between means. Before conducting f-tests of pairwise differences in averages between racial/ethnic 
group, an overall F-test was conducted of equality of averages adjusted for clustering. There was a statistically significant overall difference among the five 
racial/ethnic group for both grade 5 (F= 814.57, p < 0.001) and grade 8 (F= 830.48, p < 0.001). The p-values for the pairwise comparisons were adjusted for 
multiple testing using the Bonferroni method. 

Source: Authors' analysis based on data from the Minnesota Department of Education. 
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TABLE F2 

Percentage distributions by demographic characteristics at the student and school levels 





Student-level data 


School-level data 


Characteristic 


Grade 5 
(n = 51,510) 


Grade 8 
(n = 52,421) 


Grade 5 
(n = 786) 


Grade 8 
(n = 469) 


Female students 


49.1 


49.4 


48.6 


48.4 


Students eligible for free or reduced-price lunch 


34.6 


31.1 


43.7 


40.1 


Limited English proficient students 


3.9 


3.0 


10.2 


5.7 


Students identified for special education services 


13.3 


10.9 


14.3 


15.2 


White students 


78.6 


80.1 


71.6 


78.0 



Note: Percentages might not sum to totals because of rounding. For each grade, n is either the number of students or the number of schools. The differences 
in the percentages between the student-level and school-level data could be due to the differences in students included in each set of calculations. The 
student-level data included only students from grades 5 and 8 who were included in the analyses; the school-level data included students from all grades in 
schools that contain either grade 5 or grade 8. 

Source: Authors' analysis based on data from the Minnesota Department of Education. 



TABLE F3 

Number of limited English proficient students removed from the analyses based on missing student or 
school characteristics for grade 5 



Limited English proficient 
students with missing data 


Characteristic 


Number 


Percentage of 
original sample 
with missing data 


Original sample 


4,991 


na 


Student characteristics 


Minnesota Comprehensive Assessment-Series II (MCA— II) science scores 


96 


1.9 


Student demographics (gender, eligibility for free or reduced-price lunch, special 
education status, limited English proficiency status, race/ethnicity) 


0 


0.0 


Prior-year student achievement on MCA-II math 


2,837 


56.8 


Prior-year student achievement on MCA-II reading 


601 


12.0 


School characteristics 


Student-related (percentage of female students, students eligible forfree or 
reduced-price lunch, students identified for special education services, limited 
English proficient students, White students) 


175 


3.5 


Teacher-related (years of teaching experience, 
teacher degree attainment, student-teacher ratio) 


10 


0.2 


School average prior-year academic achievement 


181 


3.6 


Students with deleted data 


2,966 a 


59.4 b 


Final analytic sample for research questions 1 and 3 


2025 


40.6 



na is not applicable. 

Note: Percentages might not sum to total because of rounding. 

a. This number does not equal the sum of the missing school and student characteristics because some students were missing more than one characteristic. 

b. This percentage does not equal the sum of the missing school and student characteristics because some students were missing more than one 
characteristic. 

Source: Authors' analysis based on data from the Minnesota Department of Education. 
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TABLE F4 

Differences between the analytic sample of limited English proficient students used in the analyses and 
excluded students, based on Minnesota Comprehensive Assessment-Series II scores and other demographic 
characteristics for grade 5 





Analytic sample of limited 
English proficient students 


Limited English proficient students 
excluded from the analytic sample 




Characteristic 


n 


Mean 


Frequency 

(percent) 


n 


Mean 


Frequency 

(percent) 


p-value 


Achievement on 
Minnesota Comprehensive 
Assessment-Series II 
(MCA-II) science 


2,025 


534.49 


na 


2,870 


531.21 


na 


< 0.001 


Female 


2,025 


na 


46.5 


2,966 


na 


48.1 


0.26 


Eligible for free or reduced- 
price lunch 


2,025 


na 


87.6 


2,966 


na 


86.6 


0.32 


Identified for special 
education services 


2,025 


na 


13.0 


2,966 


na 


16.4 


< 0.001 


Non-White race/ethnicity 


2,025 


na 


3.5 


2,966 


na 


5.9 


< 0.001 


Prior-year achievement 
MCA-II math 


2,025 


447.11 


na 


129 


440.76 


na 


< 0.001 


Prior-year achievement 
MCA-II reading 


2,025 


444.08 


na 


2,365 


440.67 


na 


< 0.001 



na is not applicable. 

Note: T - tests were conducted to examine differences between the analytic limited English proficient sample and excluded limited English proficient sample 
for MCA— II science, MCA— II prior-year reading achievement, and MCA— II prior-year math achievement. Chi-square tests were conducted to examine differ- 
ences between the analytic limited English proficient sample and the excluded limited English proficient sample based on gender (male/female), free or 
reduced-price lunch (eligible/not eligible), special education status (identified/not identified), and race/ethnicity (White/non-White). 

Source: Authors' analysis based on data from the Minnesota Department of Education. 
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TABLE F5 

Number of limited English proficient students removed from the analyses based on missing student or 
school characteristics for grade 8 



Limited English proficient 
students with missing data 


Characteristic 


Number 


Percentage of 
original sample 
with missing data 


Original sample 


3,502 


na 


Student characteristics 


Minnesota Comprehensive Assessment-Series II (MCA— II) science scores 


65 


1.9 


Student demographics (gender, eligibility for free or reduced-price lunch, special 
education status, limited English proficiency status, race/ethnicity) 


0 


0.0 


Prior-year student achievement on MCA-II math 


1,728 


49.3 


Prior-year student achievement on MCA-II reading 


453 


12.9 


School characteristics 


Student-related (percentage of female students, students eligible for free or 
reduced-price lunch, students identified for special education services, limited 
English proficient students, White students) 


8 


0.2 


Teacher-related (years of teaching experience, 
teacher degree attainment, student-teacher ratio) 


29 


0.8 


Students with deleted data 


1,925 a 


55.0 b 


Analytic sample for research questions 1 and 3 


1,577 


45.0 


School average prior-year academic achievement 


272 


7.8 



na is not applicable. 

Note: Percentages might not sum to total because of rounding. 

a. This number does not equal the sum of the missing school and student characteristics because some students were missing more than one characteristic. 

b. This percentage does not equal the sum of the missing school and student characteristics because some students were missing more than one 
characteristic. 

Source: Authors' analysis based on data from the Minnesota Department of Education. 
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TABLE F6 

Differences between the analytic sample of limited English proficient students used in the analyses and 
excluded students, based on Minnesota Comprehensive Assessment-Series II scores and other demographic 
characteristics for grade 8 





Analytic sample of limited 
English proficient students 


Limited English proficient students 
excluded from the analytic sample 




Characteristic 


n 


Mean 


Frequency 

(percent) 


n 


Mean 


Frequency 

(percent) 


p-value 


Achievement on 
Minnesota Comprehensive 
Assessment-Series II 
(MCA-II) science 


1,577 


837.68 


na 


1,860 


833.20 


na 


< 0.001 


Female 


1,577 


na 


45.5 


1,925 


na 


46.1 


0.72 


Eligible for free or reduced- 
price lunch 


1,577 


na 


88.7 


1,925 


na 


88.2 


0.66 


Identified for special 
education services 


1,577 


na 


13.1 


1,925 


na 


14.4 


0.28 


Non-White race/ethnicity 


1,577 


na 


2.9 


1,925 


na 


7.3 


< 0.001 


Prior-year achievement 
MCA-II math 


1,577 


738.65 


na 


197 


725.05 


na 


< 0.001 


Prior-year achievement 
MCA-II reading 


1,577 


737.44 


na 


1,472 


733.41 


na 


0.006 



na is not applicable. 

Note: T - tests were conducted to examine differences between the analytic limited English proficient sample and excluded limited English proficient sample 
for MCA— II science, MCA— II prior-year reading achievement, and MCA— II prior-year math achievement. Chi-square tests were conducted to examine differ- 
ences between the analytic limited English proficient sample and the excluded limited English proficient sample based on gender (male/female), free or 
reduced-price lunch (eligible/not eligible), special education status (identified/not identified), and race/ethnicity (White/non-White); p = 0.28. 

Source: Authors' analysis based on data from the Minnesota Department of Education. 
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NOTES 




6. 


Results for limited English proficiency are not 
reported because more than half of limited 


1 . 


Full-time equivalent credits represent the 
amount of time per week a teacher is reported 
in a teaching assignment. 




English proficient students had missing data 
and were removed from the sample. Those 
that remained were not representative of all 
limited English proficient students, which 


2. 


For a review, see Flanushek (1999). 




could bias results for a comparison of limited 
English proficient and non-limited English 


3. 


Although the analyses for both of the first two 
research questions look at the relationship be- 




proficient students. 




tween gender and achievement, for instance, 
the relationships addressed differ. As Snidjers 
and Bosker (1999, p. 13-16) point out, these 
two relationships have different meanings and 


7. 


See “standardized mean differences” in box 
2 for information on the magnitude of effect 
sizes. 




the relationship between individual character- 
istics and individual outcomes can be in the 
opposite direction (as is the case here for gen- 
der) of the relationship between aggregated 
characteristics and aggregated outcomes. 


8. 


Random error includes unpredictable events 
or circumstances that can affect how students 
perform on a test. Random error can come 
from differences in how schools administer 
the test or in how students react to the test 
based on such factors as their mood, the time 


4. 


Hanushek (1999) found that lower perfor- 
mance can be associated with smaller class 




of day, and so on. 




sizes, but there are many factors that might 
affect class size and performance. For ex- 
ample, schools with many struggling or disad- 
vantaged students might reduce class size to 
accommodate these students and improve 
their achievement. These schools might have 
performed even worse if their average class- 


9. 


The percentage of students with limited Eng- 
lish proficiency had a statistically significant 
correlation. Flowever, because of the large 
amount of missing data (see box 3), results for 
limited English proficient students could be 
misleading and are thus not discussed. 


5. 


room size was larger. 

The within-school variance for Model 1 is not 
shown in table El because the focus of this 
section is on differences between schools. 


10. 


This item applies to the special education 
students who took the MCA-II. 
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